Classification in Non-Metric Spaces

A key question in vision is how to represent our knowledge of previously encountered objects to classify new ones. The answer depends on how we determine the similarity of two objects. Similarity tells us how relevant each previously seen object is in determining the category to which a new object belongs. Here a dichotomy emerges. Complex notions of similarity appear necessary for cognitive models and applications, while simple notions of similarity form a tractable basis for current computational approaches to classification. We explore the nature of this dichotomy and why it calls for new approaches to well-studied problems in learning. We begin this process by demonstrating new computational methods for supervised learning that can handle complex notions of similarity. (1) We discuss how to implement parametric met.hods that represent a class by its mean when using non-metric similarity functions; and (2) We review non-parametric methods that we have developed using nearest neighbor classification in non-metric spaces. Point (2) , and some of the background of our work have been described in more detail in [8]. 1 Supervised Learning and Non-Metric Distances How can one represent one 's knowledge of previously encountered objects in order to classify new objects? We study this question within the framework of supel vised learning: it is assumed that one is given a number of training objects, each labeled as belonging to a category; one wishes to use this experience to label new test instances of objects. This problem emerges both in the modeling of cognitive processes and in many practical applications. For example, one might want to identify risky applicants for credit based on past experience with clients who have proven to be good or bad credit risks. Our work is motivated by computer vision applications. Most current computational approaches to supervised learning suppose that objects can be thought of as vectors of numbers , or equivalently as points lying in an ndimensional space. They further suppose that the similarity between objects can be determined from the Euclidean distance between these vectors, or from some other simple metric . This classic notion of similarity as Euclidean or metric distance leads Classification in Non-Metric Spaces 839 to considerable mathematical and computational simplification . However, work in cognitive psychology has challenged such simple notions of similarity as models of human judgment , while applications frequently employ nonEuclidean distances to measure object similarity. We consider the need for similarity measures that are not only non-Euclidean , but that are non-metric. We focus on proposed similarities that violate one requirement of a metric distance , the triangle inequality. This states that if we denote the distance between objects A and B by d(A , B) , then : VA , B , C : d(A, B) + d(B, C) ~ d(A , C) . Distances violating the triangle inequality must also be non-Euclidean. Data from cognitive psychology has demonstrated that similarity judgments may not be well modeled by Euclidean distances. Tversky [12] has demonstrated instances in which similarity judgments may violate the triangle inequality. For example , close similarity between Jamaica and Cuba and between Cuba and Russia does not imply close similarity between Jamaica and Russia (see also [10]) . Nonmetric similarity measures are frequently employed for practical reasons, too (cf. [5]) . In part, work in robust statistics [7] has shown that methods that will survive the presence of outliers, which are extraneous pieces of information or information containing extreme errors , must employ non-Euclidean distances that in fact violate the triangle inequality ; related insights have spurred the widespread use of robust methods in computer vision (reviewed in [5] and [9]). We are interested in handling a wide range of non-metric distance functions, including those that are so complex that they must be treated as a black box . However, to be concrete , we will focus here on two simple examples of such distances: median distance: This distance assumes that objects are representable as a set of features whose individual differences can be measured, so that the difference between two objects is representable as a vector: J = (d1 , d2 , .. . dn ). The median distance between the two objects is just the median value in this vector. Similarly, one can define a k-median distance by choosing the k'th lowest element in this list. kmedian distances are often used in applications (cf. [9]) , because they are unaffected by the exact values of the most extreme differences between the objects . Only these features that are most similar determine its value . The k-median distance can violate the triangle inequality to an arbitrary degree (i.e. , there are no constraints on the pairwise distances between three points) . robust non-metric LP distances: Given a difference vector J, an LP distance has the form: